Evaluation of Crowdsourced User Input Data for Spoken Dialog Systems
نویسندگان
چکیده
Using the Internet for the collection of data is quite common these days. This process is called crowdsourcing and enables the collection of large amounts of data at reasonable costs. While being an inexpensive method, this data typically is of lower quality. Filtering data sets is therefore required. The occurring errors can be classified into different groups. There are technical issues and human errors. For speech recording, technical issues could be a noisy background. Human errors arise when the task is misunderstood. We employ several techniques for recognizing errors and eliminating faulty data sets in user input data for a Spoken Dialog System (SDS). Furthermore, we compare three different kinds of questionnaires (QNRs) for a given set of seven tasks. We analyze the characteristics of the resulting data sets and give a recommendation which type of QNR might be the most suitable one for a given purpose.
منابع مشابه
A Comparative Analysis of Crowdsourced Natural Language Corpora for Spoken Dialog Systems
Recent spoken dialog systems have been able to recognize freely spoken user input in restricted domains thanks to statistical methods in the automatic speech recognition. These methods require a high number of natural language utterances to train the speech recognition engine and to assess the quality of the system. Since human speech offers many variants associated with a single intent, a high...
متن کاملNatural Language Input for In-Car Spoken Dialog Systems: How Natural is Natural?
Recent spoken dialog systems are moving away from command and control towards a more intuitive and natural style of interaction. In order to choose an appropriate system design which allows the system to deal with naturally spoken user input, a definition of what exactly constitutes naturalness in user input is important. In this paper, we examine how different user groups naturally speak to an...
متن کاملDesign and Evaluation of Spoken Dialog Systems
Interactive spoken dialog systems extend the range of automated telecommunication services beyond simple limited-choice form-filling applications to goal-directed tasks covering richer, more complex domains. Creating effective and efficient dialog systems requires not only accurate ancl robust speech recognition and language modeling, but also iterative, principled design of the user interface ...
متن کاملReal user evaluation of a POMDP spoken dialogue system using automatic belief compression
This article describes an evaluation of a POMDP-based spoken dialogue system (SDS), using crowdsourced calls with real users. he evaluation compares a “Hidden Information State” POMDP system which uses a hand-crafted compression of the belief space, ith the same system instead using an automatically computed belief space compression. Automatically computed compressions re a way of introducing a...
متن کاملSpeechEval – Evaluating Spoken Dialog Systems by User Simulation
In this paper, we introduce the SpeechEval system, a platform for the automatic evaluation of spoken dialog systems on the basis of learned user strategies. The increasing number of spoken dialog systems calls for efficient approaches for their development and testing. The goal of SpeechEval is the minimization of hand-crafted resources to maximize the portability of this evaluation environment...
متن کامل